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ABSTRACT 

Two studies among U.S* college students (n«88 and 
n='329) examined the relationships between the order in which 
responses are offered on a questionnaire and the ranked importance of 
those responses. Study 1 included 36 males and 52 females, and Study 
2 included 127 males and 202 females. Both studies found that 
approximately one-third (32 percent and 31 percent, respectively) of 
listed attributes were ranked in the order of mention* Expanding the 
criteria to include adjacent categories raised the overlap to about 
two-thirds (64 percent and 65 percent, respectively). The findings of 
both studies were independent of sex. In addition, Study 2 measured 
topic importance/involvement among the respondents, and results 
indicate that it was not a factor in the trend. The factor that did 
influence the order of mention and ranking was the number of 
categories used by respondents. When more than seven categories were 
used, the stability of mention and rankings tended to become erratic. 
Three tables present study findings. (Author/SLD) 
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Abstract 

Two studies among US college students {N-| = 88; N2 = 329) examined 

the relationships between the order in which responses are offered on a 
questionnaire and the ranked importance of those responses. Both studies 
found that approximately one-third (32%; 31%) of the listed attributes 
were ranked in the order of mention. Expanding the criteria to include 
adjacent categories raised the overlap to about two-thirds (64%; 65%), The 
findings of both studies were independent of sex. In addition, Study II 
measured topic importance/involvement among the respondents; the 
results indicated it was not a factor in this trend. The factor which did 
influence the order of mention and rankings was the number of categories 
used by respondents. When more than seven categories were used, the 
stability of mention and rankings tended to become erratic. 
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Order of Elicited Responses on a Questionnaire 
as a Measure of Topic Salience 

When a person is asked the question "In your view, what are the 
essential qualities necessary for a person to be the President of the United 
States?" or *'Why do you purchase Brand X?" or "What features of an 
automobile might prompt you to consider buying one?" most respondents 
are able to generate a list of words or phrases which pollsters promptly 
record. It is legitimate to ask, therefore, whether or not the respondent 
generated list contains more information than simply the responses 
themselves. Does the order in which the items are mentioned, carry any 
impact? The tentative answer to this question appears to be yes. 

An early study (Adams-Webber & Benjafieid, 1973) found that the 
more judgmentally descriptive an adjective, the more useful that adjective 
is in describing other persons. Building on that study, McDonagh and 
Adams-Webber (1987) have recently demonstrated that the order of 
elicitation of personal constructs is related to the importance those 
constructs hold for the individual. Using Kell/s (1955) Role Construct 
Reperton/ Test (rep test) their findings displayed "no overlap at all 
between the first 5 constructs elicited from subjects and the last 5 in 
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terms of their rank-order of 'importance'" ( McDonagh & 
Adanns-Webber,1987, p. 83). While this welcome study confirms what 
might be intuitively suspected, the data are not as conclusive as it might 
seem. First, the sample used by McDonagh and Adams-Webber (1 987) 
consisted of only 1 1 females and 6 males. Second, Kelly's (1955) rep test 
is an intensely cognitive task demanding the construction of bipolar 
dimensions used to interpret and predict human behavior. Whether or not 
this type of task would generalize to issues of interest in survey research 
is questionable. Thus, the primary purpose of this study is to address 
these two central concerns. 

Assuming that a favorable answer is found to the first two issues 
above, there is an even more interesting question: Is the stability of a list 
of respondent generated attributes more or less reliable depending on the 
length of the offered list? That is, are people who respond with only two 
or three attributes ordering their responses in any better sense than the 
respondents who use nine or ten attributes? Finally, are there systematic 
differences between the responses of males and females. 

Method 

Eighty-eight US college undergraduates (36 males and 52 females) 
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served as subjects in this study. All respondents were sampled in classes 
at a mid-Atlantic land-grant university. No one was required to 
participate in the study but there were no refusals. 
Procedure 

Subjects were asked to list "those physical attributes which make 
another person attractive to you." Respondents were urged not to list any 
extraneous attributes but, by the same token, not to omit any important 
ones. Next, respondents were asked to rank order the list they had just 
created. Ail subjects were then debriefed and told the purpose of the 
study. 

The topic of physical attractiveness was deliberately selected 
because college aged persons are in the stage of life when dating and 
potential mate selection are paramount issues for them. The reasoning 
was that the generated list of attributes would provide a highly engaging 
task similar to the involvement measured by McDonagh and Adams-Webber 
(1987). 

Results 

By cross classifying the order of mention by rank it should be possible 
to determine the correspondence between the two. For example, an 
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attribute mentioned third and ranked third would be classified along the 
diagonal of a symmetrical table. As shown in Table 1 , 155 (32%) of the 
evaluations fall directly on the diagonal. While this correspondence is 
mathematically precise, it should be recognized that extemporaneous 
responses may not be so numerically pure. By expanding the acceptance 
criteria to include adjacent categories such as "near the top," e. g., the 
probability that an item listed second would be ranked first, second or 
third and, in a similar manner, by relaxing the criteria to include "near the 
bottom," e. g,, the probability that an item mentioned seventh would be 
ranked sixth, seventh, or eighth it is possible to capture the more global 
trends in the data. Using these more generous categories, the overlap 
rises to 311 mentions (64%). 



Insert Table 1 about here 



To deal with the question of stability, the absolute difference between 
each listed item and the rank of that item was summed. Understanding 
that a long list would be more unstable than a short one, this summated 
score was normed by the number of categories used by the respondent. The 
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derived formula was: Z |(List n - Rank n)|/ (number of categories used). 
The practical implication of this formula is that the smaller the observed 
coefficient, the more stable the response pattern. With that understanding 
the results of Table 2 may be examined. 



Insert Table 2 about here 



The results reveal no differences between the stability coefficients of 
males (X =1 .25; N = 36) and females (X = 1 .25; N = 52). There was a 
substantial main effect, however, which depended on the number of 
categories used in describing physical attributes of attractiveness (£ (8, 
72) = 2.48; e = .02). The interaction between sex and category number was 
not significant (F (6, 72) = .72; q = .64). 

Table 2 indicates a clear division which appears to occur between 
those who offer seven or fewer categories and those who use eight or more 
categories. This contrast is quite significant (i = 6.61; df = 18.2; q < .001). 

Discussion 

These results are fairly straightforward in one sense: there is a 
relationship between the order in which people offer free-form responses 
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of attractiveness attributes and the ranked importance of those attributes. 
!n another sense, there is less solace. There is still a substantial amount 
of error in spite of the generous categories such as "near the top" and "near 
the bottotn." Fully one-third of all responses fall outside even these broad 
classification schemes. Even so, researchers may find utility in these 
results. None but the most naive believe in a perfect world and there is 
certainly more than random chance in these findings. 

The additional analysis that seven or fewer categories provide the 
greatest amount of stability also comes as no surprise. The number seven 
is a familiar integer in the social sciences. More than a quarter century 
ago. Miller (1956) demonstrated that seven categories is the normative 
standard in information processing. The present study meshes neatly with 
that classic finding. 

In spite of the promising nature of these findings there are some 
limitations which should be noted. First, it cannot be assumed that the 
lists provided by respondents will automatically take into account the full 
spectrum of salient attributes. To put the findings into perspective, no 
respondent listed what is perhaps the most salient physical attribute of 
attraction: age (Buss, 1985). Second, a general population sample might 
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well exhibit differing patterns from those of college students. Third, we 
do not know whether or not these findings can be generalized to other topic 
areas. Finally, the topic of physical attractiveness was deliberately 
selected because college-aged persons are so intensively involved in 
dating and potential mate selection. Thus, the responses could have easily 
been influenced by the high salience of the topic. A low salient topic 
might yield very different results. 

Even with all these resen/ations, the results outlined here point to the 
faci: that: (1 ) substantial overlap seems to exist between the order in 
which items are elicited and the ranked importance of those items, (2) the 
relationship is independent of gender, and (3) the most stable of these 
responses extends up to and including, but not beyond, seven freely offered 
categories. 

Study II 

The ambiguity of study I in terms of both topic area and restricted 
personal impact prompted a replication of these provocative findings. In a 
great many aspects of everyday life a person's opinions are not always 
focused on matters of high personal salience. This recognition extends to 
such arenas as national politics, consumer purchases and a host of others. 
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For many citizens these matters are not important but they are issues 
which concern survey researchers much more than personal constructs. 
Hence, the focus of this second study will be on an issue which is not 
personally important to all respondents. The question is: are those trends 
found in the first study replicated when an issue is of low or negligible 
salience? Second, but no less important, are the trends of the first study 
evident with a different topic area? 

The topic chosen for this study was politics. As a case in point, the 
election between George Bush and Michael Dukakis witnessed less than 
one-half of the eligible electorate casting any ballot at all. Hitting a 64 
year low in voter turnout, US voters stayed home in droves. This topic 
would seem to be ideal in finding a broad range of salience among 
respondents. 

Method 

The subjects of this study were drawn from the same university as 
those for study 1. Among the 332 respondents, there was a single 
quasi-refusal. That person explained that the attributes were "all of equal 
importance" thereby failing to complete the task. Two other respondents 
had to be discarded because they made errors in the ranking process. This 
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left a usable response of 329 respondents: 202 females and 127 males. 
Prgggd.urg 

Subjects were asked, "In your view, what are the essential qualities 
necessary for a person to be the Governor of this state?" As before, the 
generated list was rank ordered afterward. In addition, subjects were 
given a nine-point scale which ranged from very low involvement to high 
involvement. Some examples are : "I don't know anything about state 
politics and I don't care" (1), "I know v^ho the really important state 
officials are but probably couldn't name the Attorney General" (3), "1 
seldom if ever vote in state/local elections. They simply don't seem all 
that important to me" (5), "I alway? vote in state/local elections and I'm, 
keenly interested in state politics" (7), and finally "I have given money 
and/or time to state political causes and/or candidates"(9). 

The effort in this Guttman-type scale was to measure incremental 
stages from total noninvolvement to high levels of political interest and 
commitment. Each respondent saw the total scale shown on an overhead 
and placed the appropriate number between one and nine on the form 
provided. All subjects were then debriefed and told the purpose of the 
study. 
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Results 

As a manipulation check the first order of business is to ascertain the 
utility of the salience scale. The scale seems to have been successful. 
The mean salience level is in the mid-range of the scale (X = 5.33; N = 329) 
with almost one-third (32%) of the respondents indicating they had never 
registered to vote in any election to over one-fifth (21%) saying that they 
alwayg (italics in the original scale) voted in state and local elections. 

The next question involves the relationships among the measured 
variables of sex, category numbers and salience. The only significant 
finding is between the number of categories used and sex (r = -.16; p < 
.003). A further examination reveals that females tend to use 
significantly more categories (X = 6.22; N = 202) than do males (X = 5.47; N 
= 1 27). On the other hand, it is now clear that the number of descriptive 
attributes used has almost nothing to do with how important state politics 
is in the lives of the respondents (r = .09; q is n. s.) but the sex-based 
finding means that separate analyses need to be undertaken for each 
gender. 

In an analysis comparable to the one shown in Table 1 , low 
involvement (unable to name the -state Attorney General, categories 1 to 3) 
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females (N = 34) elicited and ranked attributes in the same order 21% of 
the time with the expanded elicltation and rankings coming in at 54%. For 
high involvement ( always vote in same and local elections, categories 7 to 
9) females (N = 36) the same figures were 32% and 62%. Low involvement 
males (N = 27) were 33% and 61% for the same comparisons with high 
involvement males (N = 34) showing rates of 53% and 68%. 

Once more stability coefficients were calculated for each respondent 
in the study. This coefficient is a rather direct measure of closeness of 
fit between the offered list and the ranked list. The three-way analysis 
of variance to examine the effects of sex, numbers of categories used and 
importance on stability yielded a main effect only for number of 
categories used (F (15,305) = 18.40; a < ,001). This result suggests that it 
is not sex which is causing the observed differences but the tendency of 
females to utilize more categories than males. Unfortunately, the 
interaction terms could not be examined to ascertain the validity of this 
assertion because of a number of zero entry cells. 

There was, however, another way to attack the problem. The same 
variables were used in an analysis of covariance design with sex as the 
covariate. As a partial correction for the empty cell problem, some 
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adjacent response categories were collapsed into fewer cells to achieve a 
more balanced sample for category numbers used. The results revealed 
that the covariate sex was highly significant (F (1 , 273) = 10.92; q < .001) 
along with the main effect of category number (F (13, 273) = 28.76; q < 
.001). The interaction between the number of categories used and 
importance was nonsignificant. These results strongly support the notion 
that the interaction between sex and category number used is very strong. 
Thus, the more precise answer is not that males and females respond 
differently in eliciting and ranking a list of political attributes but that 
those persons offering larger numbers of categories respond differently 
than those offering fewer categories. This is the same result as found in 
study I. 

Finally, the issue of stability is once again examined. Using the 
collapsed typology for category numbers to facilitate examination of 
stability, the results appear in Table 3. 



Insert Table 3 about here 
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While some argument might ensue over where to make the decision for 
inclusion or exclusion, once again the clustering of results led to the 

4 

choice of those respondents using seven or fewer categories. These 
persons seem to be the most reliable respondents and to provide the best 
test of how much credence might be placed in judging elicited opinions. 
Moreover, we already know that they do not differ from those persons with 
larger numbers of categories on either sex or felt importance, in this 
spirit, the offered list and the ranked list overlapped in 31% of the cases, 
while the more expanded standards included 65% of the cases. Again, these 
results are amazingly similar to those of study 1. 

Conclusions 

These two studies combined seem to point to some tentative 
conclusions: 

1 . It is possible to glean more information from sun/ey respondents 
than simply a categorical list of attributes concerning some topic. 

2. The correspondence between the order in which a list of 
attributes is offered and the importance of that list of 
attributes is about one-third. That is. 33% of the freely offered 
attributes concerning a subject are ranked in importance in 
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precisely the same order as they are mentioned by respondents. 

3. If we are willing to be a bit more generous in our interpretations, 
e. g., "near the top" or "near the bottom," the correspondence 
between freely offered attributes and the rank in importance of 
those same attributes rises to about two-thirds. 

4. These findings appear to hold even in such diverse areas as the 
desirable attributes of physical attractiveness and the qualities 
necessary to be the Govarnor of a state. 

5. In addition, the findings of these studies suggest that neither the 
salience of the topic nor the sex of the respondent need be factors 
of concern for the researcher. 

6. These findings are tempered by the fact that as the number of 
freely offered responses increases beyond seven categories, the 
stability of the list decreases. Thus, it is recommended that 
these conclusions apply most directly to those persons offering 
seven or fewer responses. 

These studies have revealed a potentially new aspect of information in 
survey research and one which it is hoped will be expar.ded to more diverse 
topic areas as well as to more general population samples. 
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Table 1 

Cross tabulation of order of mention by rank^ 
Ranks 

Order 

of 1 2 3 4 5 6 7 8+ 

Mention 



1 


36 


22 


12 


12 


5 


1 


0 


0 


2 


18 


28 


20 


10 


8 


2 


2 


0 


3 


12 


17 


20 


14 


15 


4 


3 


1 


4 


12 


13 


14 


22 


12 


5 


1 


0 


5 


5 


4 


8 


14 


19 


11 


3 


1 


6 


3 


2 


8 


6 


4 


14 


5 


2 


7 


1 


2 


2 


1 


2 


5 


10 


0 


8+ 


1 


0 


0 


0 


0 


0 


0 


6 



^This is not an ordinary cross-classification table since each respondent 
appears in every row and every column depending on the number of 
categories used. 
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Table 2 

Stability Coefficients Classified by 
Number of Categories 
(Study I) 



Sex 

Number 

of Male (N) Female (N) Total 
Categories 



2 0.00 (2) 0.00 (0) 0.00 

3 1.00 (2) .67 (5) .76 

4 1.00 (4) 1.33 (9) 1.23 

5 1.60 (7) 1.07(15) 1.24 

6 1.37(10) 1.39(11) 1.38 

7 1.06 (7) 1.20 (5) 1.12 

8 1.75 (3) 1.80 (5) 1.78 

9 1.56 (1) 1.78 (1) 1.67 

10 0.00 (0) 1.80 (1) 1.80 



RESPONSE ORDER 
20 



Table 3 

Stability Coefficients Classified by 
Number of Categories 
(Study 11) 



Number of Stability 

Categories Coefficient N 

3 or fewer .51 29 

4 1.01a 59 

5 1.05a 71 

6 1.51b 57 

7 1.71b 51 

8 1 .98 32 
9 or more 2.48 30 



Letters denote categories which do not significantly differ from one 
another using the Newman-Keuls procedure 



